Summary

DRAFT / POC / WORK IN PROGRESS

Implements #178613.

This PR creates a new plugin @kbn/aiops-api-plugin. Its purpose is to host AIOps related functionality to be consumed by Kibana stack and its solutions. In contrast to that, the APIs in @kbn/aiops-plugins should be considered completely internal and are implementation details of UI components in the ML UI. The separate plugin is created to isolate these APIs so we don't run in to problems with cyclical dependencies.

This version of Log Rate Analysis at a minimum just needs an index name, a time range and the time field name of the index (TODO: Still need to add support for an optional query to limit the scope of the analysis). It will then run the following analysis:

• Change point detection runs over the time range. Based on detected spikes/dips it identifies baseline and deviation time ranges to run the analysis.
• Using field caps it identifies fields suitable for analysis. If you know the fields you want to analyse up front, you can provide them and it will skip this step.
• Significant terms with the p-value option is used to identify statistically significant keyword-like fields.
• Text categorization and custom code we also use for data drift detection is used to identify statistically significant patterns in text fields.

The analysis then returns some metadata about the change point, the significant items, as well as the date histogram used to identify the change point. This can be used then to populate custom UIs or hand it off to an LLM.

The PoC in this PR exposes Log Rate Analysis in three different ways to be used: Plain async function, Kibana REST API endpoint, O11y AI Assistant registered function. Note the naming of functions/endpoints etc. should be considered experimental/up for discussion. Feedback very welcome.

plain async function

This allows adding log rate analysis to be included in your own Kibana server side code, for example as part of a custom REST API endpoint.

/**
 * Fetches log rate analysis data.
 *
 * @param esClient Elasticsearch client.
 * @param index The Elasticsearch source index pattern.
 * @param start The start of the time range, in Elasticsearch date math, like `now`.
 * @param end The end of the time range, in Elasticsearch date math, like `now-24h`.
 * @param timefield The Elasticesarch source index pattern time field.
 * @param abortSignal Abort signal.
 * @param keywordFieldCandidates Optional keyword field candidates.
 * @param textFieldCandidates Optional text field candidates.
 * @returns Log rate analysis data.
 */
export const fetchSimpleLogRateAnalysis = async (
  esClient: ElasticsearchClient,
  index: string,
  start: string,
  end: string,
  timefield: string,
  abortSignal?: AbortSignal,
  keywordFieldCandidates: string[] = [],
  textFieldCandidates: string[] = []
) => { ... }

import { fetchSimpleLogRateAnalysis } from '@kbn/aiops-log-rate-analysis/queries/fetch_simple_log_rate_analysis';

...

const logRateAnalysis = await fetchSimpleLogRateAnalysis(
  client,
  '.ds-filebeat-8.2.0-2022.06.07-000082',
  'Jun 7, 2022 @ 03:50:26.914',
  'Jun 7, 2022 @ 10:00:31.928',
  '@timestamp',
  abortSignal,
  ['service.name'],
  ['message']
);

Kibana REST API endpoint

The REST API endpoint just wraps the function from above. Use in Kibana Dev Tools looks like this:

POST kbn:/internal/aiops/simple_log_rate_analysis
{
    "index": ".ds-filebeat-8.2.0-2022.06.07-000082",
    "start": "Jun 7, 2022 @ 03:50:26.914",
    "end": "Jun 7, 2022 @ 10:00:31.928",
    "timefield": "@timestamp",
    "keywordFieldCandidates": ["service.name"]
}

This is the response you'll get:

{
  "logRateChange": {
    "type": "spike",
    "timestamp": 1654586417118,
    "logRateChangeCount": 103491,
    "averageLogRateCount": 9266,
    "logRateAggregationIntervalUsedForAnalysis": "5 minutes",
    "documentSamplingFactorForAnalysis": 0.1,
    "extendedChangePoint": {
      "startTs": 1654586121051,
      "endTs": 1654587009252
    }
  },
  "significantItems": [
    {
      "field": "service.name",
      "value": "postgres",
      "type": "metadata",
      "documentCount": 156280,
      "baselineCount": 60680,
      "logRateChangeSort": 2.5754779169413315,
      "logRateChange": "2.58x increase"
    }
  ],
  "dateHistogramBuckets": {
    "1654566580629": 4287,
    "1654566876696": 5411,
    "1654567172763": 5215,
    "1654567468830": 4691,
...
    "1654579903644": 7708,
    "1654580199711": 7059,
    "1654588193520": 1996,
    "1654588489587": 12337,
    "1654588785654": 3082
  },
  "windowParameters": {
    "baselineMin": 1654579692171,
    "baselineMax": 1654586121051,
    "deviationMin": 1654586417118,
    "deviationMax": 1654587009252
  }
}

Some of the attributes may seem a bit verbose but that wording seems to help the AI Assistant make better sense of what it has to interpretate. The type attribute for significant items gets returned as metadata for keyword-like fields and 'log message pattern for text fields, again to give a better hint to LLMs instead of just keyword/text.

O11y AI Assistant registered function

The analysis is also available as a registered function get_aiops_log_rate_analysis for the Observability AI Assistant. It registers a custom starter prompt in Discover that will then trigger the analysis for the data on display:

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Flaky Test Runner was used on any tests changed
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility
• This was checked for breaking API changes and was labeled appropriately